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■ Abstract 

Conventional classical confidence intervals in specific cases are un- 
Qh| physical. A solution to this problem has recently been published by 

Feldman and Cousins ]l|]. We show that there are cases where the 
new approach is not applicable and that it does not remove the basic 
\ deficiencies of classical confidence limits. 
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g : 1 Introduction 

00 . 

Q\ . Feldman and Cousins propose a new approach to the computation of classi- 

cal confidence bounds which avoids the occurrence of unphysical confidence 
• i-H | regions, one of the most problematic features of the conventional classical 

confidence limits. In addition it unifies the two procedures " computation of 
confidence intervals" and " computation of confidence limits" . The unified 
treatment represents a considerable improvement compared to the conven- 
tional classical method and has already been adopted by several experiments 
and is recommended by the Particle Data Group |fj]. However, it has serious 
deficiencies. 
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2 Basic idea of the unified approach 



We consider the example of section B of Ref. 0. For a Gaussian resolution 
function P(x; ji) we define for each mean \i an interval X\ < x < X2 with the 
property 



where a is the confidence level. For a measurement x all values /x with the 
property £i(/i) < x < X2(a0 f° rm the confidence interval. The intervals have 
the property that the true values are covered in the fraction a of a large 
number of experiments. The freedom in the choice of the interval inherent 
in the relation (1) is used to avoid unphysical limits. (Usually the limits 
X\,X2 are fixed by choosing central intervals.) In case that only one limit 
can be placed inside the allowed parameter space, upper (or lower) limits are 
computed. The data and the selected level a unambiguously fix the bounds 
and whether bounds or limits are given. The probability bounds are defined 
by an ordering scheme based on the likelihood ratio. In the case of discrete 
parameters an analogous procedure is applied with some additional plausible 
conventions. The complete recipe is too complicated to be discussed in a few 
words. The reader has to consult the original publication. 

3 Objections to the unified approach 

The new approach has very attractive properties, however, there are also 
severe limitations most of which are intrinsic in the philosophy of classical 
statistics. 

3.1 Inversion of significance 

In some cases less significant data can provide more stringent limits than 
more informative data. 

As an example we present in the following table the 90% confidence upper 
limits for a Poisson distributed signal from data with no event found (n = 0) 
for different background expectations of mean b. 

The experimental information on the signal s is the same in all four cases 
independent of the background expectation since no background is present. 




(1) 
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n=0, b=0 


n=0, b=l 


n=0, b=2 


n=0, b=3 


standard classical 


2.30 


1.30 


0.30 


-0.70 


unified classical 


2.44 


1.61 


1.26 


1.08 


uniform Bayesian 


2.30 


2.30 


2.30 


2.30 



Table 1: Confidence limits for Poisson distributed data with n observed 
events and expected background with mean b. 

For the case n = 0, b = 3 the unified approach avoids the unphysical negative 
limit of the conventional classical method but finds a limit which is more 
significant than that of an experiment with no background expected and 
twice the flux. 

If in the n = 0, b = 3 experiment by an improved analysis the background 
expectation is reduced, the limit becomes worse. 

The reason for this unsatisfactory behavior is related to the violation of 
the likelihood principle?] by the classical methods. All four cases presented in 
the table have common likelihood functions L ~ e~ s of the unknown signal 
up to an irrelevant multiplicative constant depending on b. 

3.2 Difficulties with two-sided bounds 

Let us assume a measurement x = of a parameter x with a physical bound 
— 1 < x < 1 and a Gaussian resolution of a = 1.1. (This could be for example 
a track measurement by a combination of a proportional wire chamber and 
a position detector with Gaussian resolution.) The unified approach fails to 
give 68.3% confidence bounds or limits. 

3.3 Difficulties with certain probability distributions 

The prescription for the definition of the probability intervals may lead to 
disconnected interval pieces. A simple example for such a distribution is the 
superposition of a narrow and a wide Gaussian 

P(x; n) = -= {0.9 exp (-(a; - /i) 2 /2) + exp (-(ac - ^) 2 /0.02) } 
1 A detailed discussion of the likelihood principle and references can be found in H and 
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with the additional requirement of positive parameter values [i. It will pro- 
duce quite odd confidence intervals. 

Another simple example is the linear distribution 

P(x;9) = ^(l + 9x) 

where the parameter 9 and the variate x are bound by \9\ < 1 and \x\ < 1. 
(The variable x could be the cosine of a polar angle.) Values of 9 outside 
its allowed range produce negative probabilities. Thus the likelihood ratio 
which is used as a ordering scheme for the choice of the probability interval is 
undefined for \9\ > 1. Remark that also the conventional classical confidence 
scheme fails in this case. 

Similarly all digital measurements like track measurements with propor- 
tional wire chambers or TDC time registration cannot be treated. Since the 
probability distributions are delta-functions the bounds are undefined. 

3.4 Restriction due to unification 

Let us assume that in a search for a Susy particle a positive result is found 
which however is compatible with background within two standard devia- 
tions. Certainly one would prefer to publish an upper limit to a measurement 
contrary to the prescription of the unified method. 

3.5 Difficulty to use the error bounds 

Errors associated to a measurement usually are used to combine results from 
different experiments or to compute other parameters depending on them. 
There is no prescription how this can be done in the unified approach. Aver- 
aging of data will certainly be difficult due to the bias introduced by asym- 
metric probability contours used to avoid unphysical bounds. Feldman and 
Cousins propose to use the conventional classical limits for averaging. Thus 
two sets of errors have to be documented. 

3.6 Restriction to continuous variables 

It is not possible to associate a classical confidence to discrete hypothesis. 
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3.7 Subjectivity 

The nice property of a well defined coverage depends on pre-experimental 
analysis criteria: The choice of the selection criteria and of the confidence 
level as well as the decision to publish have to be done independently of the 
result of the experiment. This requirement is rather naive. 

4 Conclusions 

There are additional difficulties to those discussed above: The elimination of 
nuisance parameters and the treatment of upper Poisson limits with uncer- 
tainty in the background predictions pose problems. These may be tractable 
but certainly introduce further complications. The computation of the limits 
will be very computer time consuming in most cases. The essential objec- 
tions, however, are those mentioned in sections 3.1, 3.3 and 3.5. It is abso- 
lutely intolerable that significant limits can be obtained with poor data and 
it is also essential to have useful error intervals. Feldman and Cousins are 
aware of the difficulties related to the inversion of significance and to biased 
errors and propose to publish additional information. This certainly is a sen- 
sible advice but does not justify classical limits. Most of the deficiencies of 
the conventional classical method remain unresolved in the unified approach. 

The experimental information relative to a parameter can be documented 
by its likelihood function. The log-likelihood functions of different experi- 
ments can easily be combined without introducing biases simply by adding 
them. In most cases the likelihood function can be parametrized in a sensible 
way, as is common practice, by the parameters which maximize the likeli- 
hood and the values at 1/y/e of the maximum. The latter define an error 
interval. In the case of Poisson limits the Bayesian limits with constant prior 
(see Table 1) provide a useful parametrization which avoids the difficulties of 
section 3.1. These pragmatic procedures, however, do not allow to associate 
a certain coverage to the intervals or limits. Coverage is the magic objective 
of classical confidence bounds. It is an attractive property from a purely 
esthetic point of view but it is not obvious how to make use of this concept. 
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